Moving U p the Information Food Chairl: Deploying Softbots on the World Wide "\i\Teb

I view the World Wide Web as an information food chain (figure 1). The maze of pages and hyperlinks that comprise the Web are at the very bottom of the chain. The WebCrawlers and Alta Vistas of the world are information herbivores; they graze on Web pages and regurgitate them as searchable indices. Today, most Web users feed near the bottom of the information food chain, but the time is ripe to move up. Since 1991, we have been building information carnivores, which intelligently hunt and feast on herbivores in Unix (Etzioni, Lesh, & Segal 1993), on the Internet (Etzioni & Weld 1994), and on the Web (Doorenbos, Etzioni, & Weld 1996; Selberg & Etzioni 1995; Shakes, Langheinrich, & Etzioni 1996). computer science. But be warned, the Web community is hungry, impatient, and skeptical. They expect: .Robustness: a working system, accessible seven days a week, twenty-four hours a day. .Speed: virtually all widely-used Web resources begin transmitting useful ( or at least entertaining) information within seconds. .Added Value: any incre~lSe in sophistication had better yield a tangible benefit to users. Is the Web challenge a distraction from our longterm goal of understanding intelligence and building intelligent agents? I believe that the field benefits from a mixture of long-term and short-term goals and from both empirical and theoretical work. Work toward the goal of deploying intelli~;ent agents on the Web is a valuable addition to the current mix for two reasons. First, the Web suggestfi new problems and new constraints on existing techniques. Second, intelligent Web agents will provide tangible evidence of the power and utility of AI techniques. J~ext time you encounter AI bashing, wouldn't it be satisfying to counter with a few well-chosen URLs? Personally, I find the Web irresistible. To borrow Herb Simon's phrase, it is today's "Main Chance." Simon describes his move from the "academic backwater" of public administration to AI and cognitive psychology as "gravitating toward the sun" (Simon 1991, pages 113-L14). While AI is not an academic backwater, the Web is today's sun. Turning towards the sun and responding to the Web challenge, my collaborators and have begun to deploy a species of information carnivores (called softbots) on the Web. Motivation Today's Web is populated by a panoply of primitive but popular information services. Consider, for example, an information cow such as Alta Vista. Alta Vista requires massive memory resources (to store an index of the Web) and tremendous network bandwidth (to create and continually refresh the index). The cost of these resources is amortized over millions of queries per day. As a result, the CPU cycles devoted to satisfying each individual query are sharply curtailed. There is no time for intelligence. Furthermore, each query is independent of the previous one. No attempt is made to customize Alta Vista's responses to a particular individual. The result is homogenized, least-commondenominator service. In contrast, visionaries such as Alan Kay and Nicholas Negroponte have been advocating agents personal assistants that act on your behalf in cyberspace. While the notion of agents has been popular for more than a decade, we have yet to build agents that are both widely used and intelligent. The Web presents a golden opportunity and an implicit challenge for the AI community. As the old adage goes "If not us, then who? And if not now, when?" The challenge of deploying web agents will help revitalize AI and forge closer links with other areas of Soft hots Softbots (software robots) all! intelligent agents that use software tools and services on a person's behalf (see figure 2 for a softbot family tree). Tool use is one of the hallmarks of intelligence. In many cases, softbots rely on the same too:ls and utilities available to human computer users -tools for sending mail, printing files, and so on. Mobile robots have yet to achieve the physical analog -using vacuum cleaners,